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PREFACE 


This  report  describes  activities  executed  by  the  STEM  Center,  Florida  State  University,  Panama 
City  branch  campus,  its  contractors  and  a  group  of  partially  compensated  volunteer  teachers  in 
the  Bay  County  public  school  system  (BCSS)  during  the  2011-2012  and  2012-2103  school 
years.  The  project  officer  was  Dr  Joe  Wander,  AFRL/RXQ.  Funding  was  provided  by  the  Air 
Force  Science,  Technology,  Engineering  &  Mathematics  (STEM)  Outreach  Coordination  Office 
(AFSOCO)  through  grant  FA4819-1 1-1-5218,  awarded  in  connection  with  an  educational 
partnership  agreement  between  AFRL/RXQ  and  BCSS. 
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1.  SUMMARY 


Ignorance  about  and  even  phobias  against  science  abound  in  the  general  US  population,  and  the 
Department  of  Defense  is  investing  substantial  sums  of  money  to  rekindle  student  interest  in 
careers  in  technical  areas  (science,  engineering,  technology  and  mathematics — STEM).  Several 
factors  contribute  to  each  of  these  situations  but  a  prominent  factor  they  share  is  the  absence  of 
quantitative  content  in  elementary  school  classrooms — the  educational  window  in  which  many 
lifelong  attitudes  are  formed. 

Seeking  to  “measure”  this  problem  and  to  evaluate  the  effect  of  systematic  intervention  in  the 
fonn  of  teacher  enrichment,  the  Air  Force’s  STEM  program  at  Tyndall  Air  Force  Base  (AFB) 
undertook  a  program  with  the  Northwest  Florida  Regional  STEM  Center,  located  within  the 
Panama  City  branch  campus  of  Florida  State  University  (FSU-PC),  to  track  and  compare  scores 
on  standardized  tests  (administered  at  intervals  to  evaluate  student  progress)  in  two  groups  of 
grade  4,  5,  6  and  7  classrooms  that  included  a  number  of  Title  1  students — a  set  of  controls  who 
followed  the  standard  curriculum  and  a  test  group,  whose  teachers  1)  before  and  during  the 
2011-2012  school  tenn  (SY  2011-2012)  participated  in  two  weekends  of  hands-on  laboratory 
experiences  guided  by  a  subject-matter  expert  (SME)  who  also  developed  lesson  plans  to  which 
the  lab  experience  was  relevant  and  2)  received  an  aquarium,  a  digital  microscope,  minor 
equipment  items  and  supplies,  and  a  kick  net  with  which  each  collected  coastal  marine  biota  to 
populate  the  aquaria. 

Several  uncontrolled  factors  complicated  the  execution  of  the  project.  Turnover  of  both  students 
and  teachers  in  Title  1  classrooms  was  high,  and  both  eroded  the  n  for  the  data  set.  The  initial 
contracting  strategy  was  processed  at  length  and  eventually  rejected,  which  delayed  the  start  of 
preparations  and  dissipated  the  initial  enthusiasm  of  the  participating  teachers.  The  SME 
commuted  from  the  upper  Midwest  for  the  weekend  sessions  and  was  not  available  on  call;  only 
part  of  that  gap  was  covered  by  middle-  and  high-school-science  master  teachers — direct  contact 
time  was  insufficient.  And  strains  deriving  from  a  combination  of  factors  eventually  led  to 
departure  of  the  SME  and  master  teachers  from  the  end  of  the  second  training  session  to  the  end 
ofSY  201 1-2012. 

A  new  master  teacher  was  recruited  during  SY  2012-2013  but  the  combination  of  loss  of 
corporate  memory  and  talent  with  the  departure  of  many  critical  personnel  and  students  made  it 
impractical  to  attempt  to  continue  the  original  enrichment  exercise,  so  the  experiment  was  not 
continued  during  the  new  term  as  a  measurement  of  residual  effect  on  the  participating  teachers’ 
new  classes  and  students  advancing  from  the  participating  classrooms. 

The  missteps  during  startup  were  one-time  events  at  the  site  but  are  a  risk  to  recur  at  new 
locations,  and  should  be  identified  as  lessons  learned  and  not  to  be  repeated;  however,  the 
intrinsic  instability  of  Title  1  students  and  classrooms  is  relatively  constant  and  will  degrade  the 
efficiency  of  local  efforts  to  effect  such  enrichment  unless  and  until  it  is  implemented  nationally. 
However,  that  education  in  Title  1  schools  is  intrinsically  limited  by  extracurricular  distractions 
limits  the  efficiency  of  such  a  STEM  promotion  effort  and  raises  a  question  about  the  purpose  of 
that  effort.  Whereas  the  social  value  to  Title  1  programs  initially  appears  obvious,  the  return  in 
the  form  of  increasing  the  pool  of  students  opting  for  STEM  careers  might  be  greater  if  the 
investment  were  directed  at  a  more-secure  economic  class. 
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2.  INTRODUCTION 


The  Department  of  Defense  (DoD)  and  a  variety  of  other  organizations  are  investing  in  a  number 
of  innovative  programs  aimed  at  drawing  a  larger  number  of  young  US  citizens  into  careers  in 
various  fields  of  engineering,  and  into  supporting  areas  of  science  and  mathematics,  under  a 
broad  umbrella  of  science,  technology,  engineering  and  mathematics  (STEM)  activities.  The 
most  dramatic  and  visible  STEM  activities,  e.g.,  Project  Lead  the  Way  (PLTW),  have  been 
directed  primarily  at  high-school  and  secondarily  at  middle-school  age  groups,  presumably  in 
proportion  to  their  capacity  to  conduct  elaborate  projects.  However,  preferences  of  subjects  and 
most  personal  attitudes  are  already  well  established  by  the  end  of  middle  school,  and  math  and 
science  phobias  are  firmly  engrained.  A  consequence  of  this  is  widespread  ignorance  and 
irrational  and  inappropriate  fear  of  science  and  technology — a  failing  of  the  educational  system 
that  leaves  the  public  vulnerable  to  exploitation  by  intimidation  and  disinformation  disseminated 
through  both  traditional  and  social  media. 

When  this  project  was  undertaken  Bay  County  (Florida)  was  the  site  of  several  active, 
engineering-oriented  STEM  projects  aimed  primarily  at  students  in  the  higher  grades,  and 
science  scores  by  the  county’s  public  school  students  in  elementary  grades  have  had  a  long 
history  of  ranking  close  to  the  national  average  at  the  end  of  grade  3  (science  is  introduced  in 
grade  4),  falling  drastically  at  the  end  of  grades  4  and  5,  and  recovering  only  gradually  in  later 
grades.  The  curriculum  that  trains  primary  and  middle  school  teachers  in  Florida  (and  generally 
throughout  the  country)  emphasizes  method  over  content,  and  math  and  science  courses  are 
generally  only  electives  in  a  crowded  environment  of  coursework  biased  toward  development  of 
language  skills — i.e.,  the  group  that  teaches  elementary  school  grades  appears  to  functionally 
select  for  math  and  science  phobics. 

Teacher  enrichment  programs  are  typically  studied  through  measures  of  progress  of  the  teachers, 
whereas  a  metric  of  at  least  equal  importance — particularly  to  a  STEM  project — would  be  the 
students’  performance  and  career  decisions.  Bay  County’s  problem  with  science  test  scores  has 
been  recognized  for  decades  and  has  been  attacked  with  earlier  local  projects,  of  which  some 
temporarily  showed  anecdotal  positive  results  from  enrichment  exercises,  but  the  successes  and 
lessons  learned  using  this  potentially  valuable  metric  were  not  documented  and  have  since  been 
lost.  This  project  is  an  effort  to  implement  a  systematic  enrichment  of  science  content  by  expert 
development  and  presentation  of  classroom  and  field  exercises  to  a  subset  of  the  county’s  grade 
and  early  middle  school  teachers  in  vertical  columns  of  schools  feeding  two  of  the  county  high 
schools,  and  to  track  and  compare  average  scores  of  students  in  participating  classrooms  to  those 
of  students  in  nonparticipating  classrooms  for  several  years.  When  sufficient  data  are  in  hand  the 
eventual  goal  is  to  expand  the  science  content  and  preparation  in  the  elementary  education 
training  curriculum  to  institutionalize  the  advances  realized  in  the  participating  classrooms. 
More-useful  data  are  expected  to  evolve  in  subsequent  years,  when  the  dimensions  of  the  ripples 
several  years  downstream  have  been  mapped  and  when  the  participating  teachers  have 
assimilated  experience  in  applying  their  new  skills  in  the  classroom. 

It  is  encouraging  to  note  that  since  the  inception  of  this  project  PLTW  has  announced  that  they 
will  expand  their  programs  into  the  elementary  schools. 
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3.  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 


The  assumptions  on  which  this  project  were  based  are  few  and  simple:  that  any  classroom  of  25 
randomly  assigned  students  within  a  grade  in  a  single  community’s  school  system  is  a  large 
enough  group  to  be  considered  representative  of  the  population  served  by  that  school,  and  that  a 
comparison  of  trends  in  scores  on  standardized  achievement  tests  by  participating  and 
nonparticipating  (control)  classes  in  the  same  school  will  provide  a  quantifiable  metric  for  each 
grade  involved.  The  weakest  assumption — imposed  by  the  limited  scale  of  the  exercise — is  that 
differences  in  the  innate  abilities  of  the  teachers  will  not  be  a  confounding  factor  in  the 
comparisons. 

3.1.  Project  Organization 

Bay  County’s  public  school  system  (BCSS)  includes  a  small  cadre  of  “master  teachers”:  science 
teachers  in  the  higher  grades  who  invest  time  and  effort,  first  to  earn  the  master  teacher 
designation  and  then  to  share  their  experience  with  other  teachers  in  the  school  system  as 
coaches,  mentors,  consultants — whatever  is  appropriate  to  the  individual  interaction.  One  master 
science  teacher  at  each  of  two  BCSS  urban  high  schools  volunteered  to  be  at  the  top  of  a 
vertically  aligned  column  of  feeder  schools  and  mentor  a  pair  of  teachers  in  grades  4,  5,  6  and  7 
of  lower  schools  feeding  students  to  their  respective  high  schools.  Both  vertical  columns 
included  schools  with  significant  populations  of  Title  1  students.  During  the  2011-2012  school 
term  each  master  teacher  was  assigned  two  participants  in  each  of  the  four  lower  grades;  the 
following  year  a  single  (new)  master  teacher  was  emplaced. 

3.2.  Planning  and  Contracting 

During  the  course  of  the  planning  and  contracting  phases  of  this  project,  Florida  State  University, 
Panama  City  Campus  (FSU-PC),  achieved  designation  as  the  area  center  for  STEM  activities. 

The  evolution  of  the  project  began  with  identification  of  Prof  (Retired)  Wynne  Lewis,  a 
microbial  ecologist  and  respected  educational  consultant,  as  a  subject  matter  expert  (SME)  to 
develop  and  deliver  content  and  to  provide  local  support,  primarily  to  ensure  efficient 
transmission  to  the  master  teachers,  who  carried  day-to-day  responsibility  for 
coaching/mentoring/guiding  the  teachers  in  their  respective  vertical  queues. 

The  initial  contracting  strategy  was  to  make  separate  awards  to  Prof  Lewis  and  to  FSU-PC,  to 
generate  and  deliver  content  and  to  provide  supplies,  facilities  and  modest  stipends  to 
participating  teachers,  respectively.  Data  collection,  analysis  and  technical  reporting  were  to 
have  been  a  shared  task  overseen  by  Prof  Lewis.  After  numerous  documents  had  been  generated 
and  messages  exchanged  with  the  contracting  office  it  was  determined  that  a  sole-source  award 
to  FSUPC  would  be  possible  but  that  an  award  to  an  educational  consultant’s  would  require  a 
full  and  open  competition,  which  was  impractical — enough  time  had  been  invested  at  the  time 
this  decision  came  down  that  a  competitive  process  would  have  delayed  the  project  start  date 
beyond  the  start  of  the  2011-2012  school  year.  To  keep  to  a  workable  project  timeline  a 
surrogate  contracting  officer’s  technical  representative  (COTR)  was  appointed  and  the  strategy 
amended  to  a  single  award  to  FSU-PC,  who  subcontracted  Prof  Lewis  as  a  part-time  employee. 
Time  pressure  forced  us  to  this  expedient,  which  included  a  change  that  proved  to  have  profound 
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operational  consequences — shifting  control  of  the  project  from  the  SME  to  the  FSU-PC  program 
manager  (PM). 

Concurrent  with  the  year  of  contracting  vicissitudes,  a  series  of  organizational  meetings  of 
master  teachers,  the  PM  and  functional  COTR  were  held,  generally  in  FSU-PC’s  facility.  These 
included  inputs  from  Prof  Lewis,  either  electronically  or  in  person.  The  recurring  revisions  that 
eventually  accomplished  a  legally  sufficient  correction  of  the  poorly  conceived  contracting 
process  delayed  the  availability  of  funds  for  the  teachers  to  “make  things  real,”  with  the 
unfortunate  consequence  of  losing  much  of  the  initial  enthusiasm  and  momentum.  A  key  step  in 
the  eventual  awarding  of  the  grant  to  FSUPC  was  establishment  of  Educational  Partnership 
Agreements  between  the  local  Air  Force  organization  (AFRL/RXQ)  and  FSU-PC  and  the  Bay 
County  school  system — which,  in  hindsight,  should  have  been  the  first  step.  A  grant  was  finally 
awarded  to  FSU-PC  in  August  201 1,  at  the  eleventh  hour  for  implementation  for  SY  2011-2012. 

3.3.  Implementation 

3.3.1.  2011-2012  School  Year 

Bay  County  is  a  coastal  area  rich  with  extensive  estuarine  environments.  To  capitalize  on  this 
asset — both  scientific  and  contextual — each  participating  classroom  was  given  an  aquarium  and 
a  digital  microscope  to  provide  access  to  local  fauna  as  relevant  objects  for  long-term  study. 
Supplies  for  the  year  were  provided  as  well,  and  will  continue  to  be  so  for  the  anticipated 
duration  of  this  multi-year  exercise.  (Appendix  A  is  a  complete  list  of  equipment  and  supplies 
provided  to  each  teacher.)  These  were  installed  in  the  classrooms  and  a  weekend  training  session 
in  mid-September  was  conducted  by  Prof  Lewis,  both  for  and  with  the  help  of  the  master 
teachers,  and  attended  by  a  bit  more  than  half  of  the  participating  teachers.  This  exercise  was 
conducted  to  familiarize  the  teachers  with  the  new  hardware  and  to  give  them  some  supervised 
hands-on  experience  with  both  the  equipment  and  its  application  in  a  series  of  experiential 
exercises  included  in  a  lesson  plan  outline  prepared  by  the  SME.  Master  teachers  and 
participating  teachers  in  attendance  received  a  modest  stipend  to  compensate  them  for  their 
weekend  time.  A  second  hands-on  workshop,  on  methods  for  isolation  and  identification  of 
DNA,  at  the  end  of  March  2012  (and  two  makeup  sessions  in  mid-April  to  accommodate 
teachers’  schedules)  completed  the  formal  presentations  by  Prof  Lewis.  Materials  were  provided 
to  the  teachers  during  the  two  sets  of  workshops. 

A  hardware  item  provided  to  the  first  year’s  participants  was  a  simple  kick  net,  which  was  used 
in  an  initiating  group  exercise  among  the  participating  teachers,  who  met  on  another  weekend 
and  waded  out  into  a  brackish  habitat  to  collect  specimens  to  populate  their  aquaria. 

The  device  to  evaluate  the  benefit — if  any — of  this  intervention  was  to  be  a  comparison  of 
student  scores  on  a  standardized  science  test  given  at  the  end  of  grades  3,  5  and  8.  In  contrast  to 
some  earlier  middle-  and  high-school  studies,  improvement  in  student  perfonnance  rather  than  a 
direct  test  of  assimilation  of  the  content  of  the  enrichment  was  to  be  the  focus  of  the  evaluation. 
Anonymized  average  scores  for  participating  and  matched  nonparticipating  classes  at  each  of  the 
schools  were  to  be  compared. 
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3.3.2.  2012-2013  School  Year 

The  defection  near  the  end  of  the  school  year  of  both  the  SME  and  the  master  teachers 
responsible  for  execution  of  the  SY  2011-2012  program  effectively  prevented  either  a 
continuation  of  the  first  year’s  topics  or  the  introduction  of  a  new  topic  area  to  continue  the 
active  exercise.  However,  a  cross-fertilization  experiment  described  in  Appendix  C  was 
conducted  as  a  6-week  summer  internship  joining  an  elementary-school  teacher,  a  rising  Title  1 
high-school  senior,  and  a  graduate  student  from  a  historically  black  university  (HBCU)  as  a  team 
to  execute  a  “real”  experimental  task. 
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4.  RESULTS  AND  DISCUSSION 


4.1.  2011-2012  School  Year 

Of  the  16  teachers  who  undertook  participation  for  SY  2011-2012,  only  four  were  in  place  at  the 
start  of  SY  2012-2013.  The  number  of  students  still  in  place  in  the  classrooms  at  the  end  of  SY 
2101-2102  was  not  recorded,  and  student  turnover  was  not  tracked,  so  the  limited  testing  results 
(5th  grade  only),  which  suggested  slight  improvement,  cannot  be  considered  a  reliable  indicator 
of  effect. 

By  the  end  of  SY  2101-2102  the  SME  and  all  of  the  master  teachers  had,  for  a  variety  of 
reasons,  left  the  project.  A  weakness  of  the  initial  design  that  quickly  became  clear  was  that  the 
SME  traveled  from  a  remote  location  and  so  was  not  physically  available  to  participating 
teachers  on  a  day-to-day  basis.  However,  the  most  disruptive  single  factor  was  an  unworkable 
initial  contracting  strategy,  which  was  pursued  to  a  fairly  advanced  stage  of  implementation 
before  the  contracting  office  declared  that  it  faced  legal  impediments  that  would  have  taken 
another  year  to  remedy — frustration  from  the  futile  expenditure  of  enthusiastic  planning  energy 
had  reached  toxic  levels  by  the  time  the  grant  was  finally  emplaced  and  the  project  started. 

4.2.  2012-2013  School  Year 

Implementation  of  the  contracting,  logistical  and  management  architecture  was  in  place  and  a 
number  of  lessons  to  be  learned  were  in  evidence  by  the  end  of  SY2 10 1-2 102.  Contracting  for 
year  2  was  accomplished  without  event;  however,  the  intensity  of  the  effort  to  execute  the 
technical  component  of  the  program  displaced  planning  and  the  defection  of  the  perfonning  team 
late  in  the  year  removed  elements  that  would  have  been  necessary  to  deliver  the  originally 
intended  planned  content.  A  new  master  teacher  was  recruited  and  additional  equipment  and 
supplies  were  provided  to  the  participating  classrooms. 

Two  categories  of  data  were  to  be  collected  at  the  beginning  and  end  of  this  second  year  to 
measure  effects  carried  forward  from  the  first-year  teacher  preparation:  perfonnance  by  the  next 
class  taught  by  each  of  the  participating  teachers,  and  perfonnance  by  students  advancing  from 
the  participating  classes.  Interpretation  of  the  data  in  the  first  category  would  provide  a 
straightforward  measure  of  retention  and  assimilation  by  the  individual  teachers,  whereas  their 
performance  during  SY  2013-2014  would  be  influenced  by  expected  preparation  in  the  second 
round  of  teacher  preparation. 

Evaluation  of  advancing  student  perfonnance  was  to  be  somewhat  simplified  because  the  control 
group  for  comparison  was  to  be  classmates  who  did  not  come  from  participating  classrooms,  and 
anticipated  initial  test  results  were  also  to  provide  an  additional  measure  of  the  previous  year’s 
effect.  The  comparison  of  scores  by  students  advancing  from  one  participating  classroom  to 
another  was  not  expected  to  be  able  to  sort  the  effects  of  retention  by  the  teacher  and  by  the 
students. 

However,  another  series  of  lessons  surfaced  here.  The  terms  of  a  grant  typically  do  not  include  a 
requirement  for  interim  reporting,  so  the  loss  of  75%  of  the  participating  teachers  was  not 
identified  until  SY  2102-2103  was  underway  and  it  had  become  clear  that  the  body  of 
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participants  was  now  too  small  to  meaningfully  continue  the  planned  activities  for  the  tenn.  The 
important  lesson  that  was  so  concealed  until  report  preparation  was  begun  is  that  some 
verification  of  clear  understanding  on  the  part  of  participants  at  all  levels  in  the  performing  group 
of  the  specific  aims  of  the  exercise  and  how  they  are  to  be  realized. 

Standardized  testing  within  BCSS  also  proved  to  be  much  less  available  than  initial  appearances 
had  suggested.  Future  exercises  of  this  sort  will  have  to  identify  specific  testing  regimens  and 
incorporate  them  into  the  initial  design.  Use  of  a  contract  rather  than  a  grant  offers  regular  feed¬ 
back  in  the  fonn  of  interim  reports — which  provides  opportunities  for  on-the-fly  interventions — 
and  appears  a  better  mechanism  to  award  funds  for  projects  that  include  quantitative  metrics; 
however,  occasional  onsite  inspections  appear  to  be  advisable,  particularly  during  early  mid  SY. 

4.3.  General  Considerations 

From  the  perspective  of  promoting  equitability  through  social  programs,  Title  1  elementary  and 
middle  schools  appear  to  be  an  obvious  target  for  exercises  in  enrichment,  STEM  and  otherwise; 
likewise,  the  teaching  staff  in  these  institutions  might  be  expected  to  be  the  most  responsive  to 
such  reinforcement  and  access  to  teaching  tools,  so  the  concept  is  attractive  from  two 
perspectives. 

However,  1)  unless  and  until  such  opportunities  reach  a  majority  of  urban  Title  1  classrooms, 
high  rates  of  student  turnover  (i.e.,  only  partial  exposure)  will  limit  the  benefit  to  these  students 
at  more-advances  grade  levels;  2)  as  a  laboratory  for  quantitative  evaluation  of  the  effect  of 
enrichment,  these  schools  are  far  less  than  ideal  for  several  reasons — students  are  distracted  by 
events  and  uncertainties  in  their  lives  and  often  switch  schools  in  mid  tenn,  crises  are  a  common 
occurrence,  methods  of  teacher  evaluation  and  reward  appear  to  discriminate  against  Title  1 
school  staff  and,  not  surprising,  teacher  turnover  is  fairly  high.  Perfonnance  data  are  therefore 
noisy  and  the  n  of  test  populations  tends  to  fall  off  rapidly,  limiting  opportunity  for  long-tenn 
evaluation — the  true  test  of  effectiveness — of  an  exercise  involving  a  workable  number  of 
subjects;  3)  these  students  are  clearly  targets  for  attention  but  promotion  of  life  and  survival 
skills  (including  mathematics)  may  be  of  greater  value  to  them. 

The  reason  that  STEM  occupations  are  not  more  popular  is  simple  economics.  A  spectrum  of 
occupations — conspicuous  examples  include  law  (historically),  finance  and  medical 
specialties — provide  much  greater  and  more  immediate  financial  opportunities  than  science  or 
even  engineering  careers  and  attract  away  much  of  the  pool  of  domestic  talent  that  might 
otherwise  opt  for  STEM  training.  The  hands-on  experience  provided  by  robot  perfonnance 
competitions  in  intermediate  to  advanced  grades  is  an  inspired  effort  to  set  the  hook  to  draw 
students  into  engineering  and  related  fields.  However,  as  it  is  a  long-term,  spare-time  exercise 
requiring  volunteer  engineering  experience,  such  activities  likewise  benefit  primarily  the  social 
classes  that  enjoy  the  availability  of  stability  and  both  necessary  resources. 

Keeping  that  thought  and  returning  to  the  present  project,  the  true  role  of  such  an  exercise  has  to 
be  considered.  If  the  single  target  is  to  increase  the  pool  of  talent  entering  STEM  fields,  the 
complications  and  inefficiencies  plaguing  Title  1  education  severely  depress  the  cost-benefit 
equation  for  that  pool,  so  greater  impact  would  be  expected  from  making  the  investment  in 
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average,  particularly  small-town  and  rural  classrooms.  On  the  other  hand,  a  moderately  effective 
incorporation  of  science  content  in  Title  1  classrooms  should  demystify  science  and  technology 
to  these  students  as  adults,  and  might  cut  slightly  into  the  influence  of  fundamentalist  antiscience 
in  the  country — each  a  benefit  fondly  to  be  wished  for  but  practically  impossible  to  evaluate. 

The  SY  2011-2012  exercise  provided  a  rich  harvest  of  lessons  about  the  pitfalls  available  to 
federal  projects  attempting  to  measure  the  influence  of  STEM  content  early  in  the  educational 
experience,  when  attitudes  about  math  and  science  are  fonned.  SY  2012-2013  was  a  fallow 
season,  allowing  a  window  for  recovery  from  the  errors  and  disappointments  of  the  first 
experiment,  but  not  providing  the  anticipated  opportunity  to  look  for  evidence  of  a  persisting  and 
possibly  growing  effect  from  assimilation  and  application  of  the  SY  2011-2012  investments. 
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5.  CONCLUSIONS 


Both  the  design  and  the  execution  of  this  exercise  included  a  number  of  serious  faults  that 
became  evident  as  the  exercise  progressed.  Candidate  remedies  to  be  tested  during  the  next 
iteration  of  this  project  are  listed  below: 

•  Student  and  teacher  turnover  in  urban  Title  1  classrooms  is  large — enough  in  this  case  to 
drop  the  number  of  data  points  below  the  threshold  for  significance  by  the  end  of  the  first 
year.  Some  improvement  may  be  realized  in  a  vertically  integrated  k-12  system  outside 
the  city,  and  such  an  institution  is  being  recruited  for  the  next  SY. 

•  One-on-one  interaction  with  the  teachers  is  known  to  be  an  important  factor  in 
enrichment  exercises.  The  only  occasional  visits  by  a  remotely  located  SME  imposed  a 
serious  limitation  on  the  amount  of  this  type  of  exchange,  and  master  teacher  defections 
aggravated  this  deficit.  The  exercise  for  SY  2103-2104  has  recruited  a  local  entomologist 
on  FSU-PC’s  faculty  as  the  SME,  and  the  original  five  local  master  teachers  within  the 
BCSS  family  will  be  replaced  by  a  more-qualified  resource  person  recently  graduated 
from  a  doctoral  program  in  education  and  employed  by  FSU-PC.  How  the  gain  in 
experimental  skill  will  balance  against  the  loss  of  proximity  and  available  time  is  an 
element  that  will  be  evaluated. 

•  The  award  mechanism  for  SY  2103-2104  will  be  a  contract,  which  will  provide  more- 
timely  feedback  on  progress  and,  more  importantly,  on  bumps  encountered  as  the  activity 
proceeds.  Some  minimally  intrusive  mechanism  for  onsite  inspection  of  the  process  in 
action  will  be  developed  with  staff  of  the  participating  school. 

•  The  testing  mechanisms  used  to  evaluate  effects  on  student  performance  of  participation 
will  be  negotiated  with  the  school  staff  to  ensure  that  expectations  are  understood  and 
realized.  Likewise,  the  specific  enrichment  content  will  be  negotiated  with  the  school  to 
better  match  the  enhancement  to  the  school’s  capabilities  and  resources. 

After  two  years  the  concept  of  measuring  the  effect  of  enhancing  STEM  content  in  elementary- 
level  classrooms  continues  to  appear  sound  despite  initial  faults  in  the  design,  missteps  during 
initiation  and  execution,  attrition  of  the  teacher  cohort  and  emigration  of  students  from  the 
school  district.  The  project  was  originally  intended  to  continue  to  evaluate  effectiveness  through 
to  a  “true”  endpoint  by  following  the  student  cohort  to  the  stage  that  college  majors  are  declared; 
however,  given  the  rate  at  which  curricula  evolve,  the  eventual  infonnation  would  be  so  dated 
that  the  burden  of  enlisting  parents  and  developing  devices  to  preserve  participant  anonymity 
would  likely  not  be  justifiable  unless  voluntary  continuation  through  a  friending  or  similar 
interaction  in  some  social  medium  is  implemented. 

The  practical  benefit  from  intervention  by  providing  experiential  equipment  to  16  teachers  for 
enrichment  of  their  classrooms  is  not  diminished  by  the  inability  of  this  experimental  design  and 
team  to  capture  a  measure  of  its  effect,  and  it  will  exert  a  lasting  effect.  The  challenge  for  the 
next  cycle  is  to  acquire  enough  control  of  the  experimental  environment  to  extract  interpretable 
quantitative  infonnation  without  suppressing  teacher  and  (as  a  consequence)  student  growth. 
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6.  RECOMMENDATIONS 


The  concept  and  general  approach  of  measuring  trends  in  student  scores  on  interval  perfonnance 
tests  still  appear  sound,  whereas  the  implementation  fell  short  of  expectations  in  several  aspects. 
If  circumstances  allow  the  project  should  be  continued,  ideally  long  enough  to  look  for  an  effect 
on  the  number  of  students  continuing  their  education  through  college  and  on  the  college  majors 
selected  by  them — which  bears  directly  on  the  goals  of  the  STEM  initiative.  To  do  so  would 
require  individual  agreements  with  parents  to  provide  that  information,  which  was  not  part  of  the 
activities  to  date,  but  social  media  sites  suggest  a  possible  mechanism  for  achieving  such  a  long¬ 
term  evaluation. 

Strength  of  eventual  conclusions  from  continuation  of  this  study  can  and  should  be  increased  by 
conducting  parallel  such  exercises  in  different  areas  of  the  country  and  pooling  the  results.  A 
preliminary  indication  is  that  mathematical  exercises  involving  observations  from  experiential 
science  activities  could  beneficially  be  introduced  into  the  preparation  of  elementary  education 
teachers.  Continued  gathering  of  data  will  be  necessary  to  verify  that  conclusion. 

In  the  context  of  this  project,  addition  of  a  member  of  the  BCSS  (and  other  school  districts  if  the 
study  is  expanded)  office  who  deals  with  standardized  test  scores  to  provide  direct  access  to  data 
and  statistical  evaluation  of  them  is  being  strongly  recommended.  Likewise,  engagement  of  a 
BCSS  staff  member  trained  in  experimental  educational  projects  would  also  increase  the 
likelihood  of  a  subsequent  project’s  delivering  an  interpretable  data  set  of  student  scores. 
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Appendix  A:  List  of  Equipment  Provided  to  2011-2012  Participating  Classrooms 

Digital  microscope  (Figure  A-l)  and  associated  firm/software 
USB  drive 

Aquarium  (Figure  A-2),  including  pump  and  supplies 
Kick  net  (to  populate  the  aquarium) 


Figure  A-l.  Digital  Microscope 


Figure  A-2.  Aquarium 
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Appendix  B:  List  of  Equipment  Provided  to  2012-2013  Participating  Classrooms 

Triple-beam  balance 
Metric  weight  set 

Refractor  (to  measure  refractive  index) 

Commercial  education  kits:  GEMS  Color  Analyzer  and  Optics  (elementary) 

Investigating  Measurement  and  Density  (middle) 

Natural  selection  for  AP  Biology  (high  school  master  teachers)) 

Associated  supplies  were  also  provided,  and  supplies  for  2011-2012  equipment  were  replaced. 
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Appendix  C:  Summer  (2012)  Team  Internship  Project 


C.l.  Concept  and  Environment 

The  vicissitudes  during  the  process  of  initiating  a  contractual  relationship  with  FSU-PC  created  a 
singular  opportunity  in  which  funding  sufficient  to  support  a  summer  internship  project  was  at 
risk  of  expiring.  This  created  an  unplanned  window  to  explore  a  concept  of  collective  enrichment, 
teaming  students  and  teachers  to  perform  a  defined,  common  experimental  task  that  could  be 
completed  in  a  window  of  six  weeks.  An  experimental  introduction  and  supervision  were  to  be 
provided  by  an  AFRL  researcher  in  whose  area  of  specialization  and  interest  the  task  was 
defined.  The  team  comprised  a  graduate  student,  a  relatively  young  elementary  education  science 
teacher  assigned  to  a  Title  1  school  and  a  rising  high  school  senior,  also  a  Title  1  student.  The 
expectation  was  that  the  graduate  student  would  develop  communication  skills  by  organizing  the 
team’s  approach  to  the  project;  the  teacher  would  acquire  a  hands-on  perspective  on  the  research 
process  and  a  good  understanding  of  the  technical  aspects  of  the  project  to  provide  a  general 
feeling  for  biological  science;  and  the  rising  senior  would  receive  an  introduction  to  scientific 
thinking  and  lab  methods,  and  a  sense  of  the  role  of  science  in  the  world. 

C.2.  The  Lab  Task 

In  recent  years,  biofuels  have  surfaced  as  a  practicable  partial  solution  to  the  global  need  for 
more-sustainable,  “environmentally  friendly”  fuel  sources.  As  one  of  the  largest  single 
consumers  of  liquid  fuel  and  electricity,  the  US  government  is  supporting  research  into  biofuels 
as  a  possible  means  to  decrease  dependence  on  imported  petroleum  and  non-renewable  fossil 
fuels.  A  number  of  available  sources  of  biomass  from  which  power  and  fuel  can  be  produced 
have  been  examined;  one  promising  candidate,  which  enjoys  the  positive  features  of  rapid 
growth  rate,  high  energy  content  and  ubiquitous  growth  potential,  is  algae.  One  paper  reported 
that  adding  fine  particles  of  cerium  oxide  (ceria)  to  an  algal  system  1)  promotes  growth,  2) 
confers  ultraviolet  protection  on  the  cells  and  3)  improves  the  subsequent  combustion  process  of 
the  algae  as  a  fuel  additive.  This  suggested  a  compact  project — design,  conduct  and  analyze  a 
series  of  experiments  varying  the  amount  and  particle  size  of  ceria  particles  added  to  achieve  the 
fastest  rate  of  accumulation  of  algal  biomass  and,  ultimately,  the  largest  energy  yield  available 
under  those  sets  of  conditions.  As  conceived  this  entailed  growing,  collecting,  drying  and 
weighing  batches  of  algal  cells,  and  burning  them  as  a  slurry  in  diesel  fuel  in  a  calorimeter  to 
measure  the  energy  yield. 

C.3.  Preparation  for  and  Initiation  of  the  Project 

Laboratory  space  in  AFRL/RXQ’s  energy  laboratory  at  Tyndall  AFB  was  set  aside  to  house  this 
project;  the  cognizant  research  engineer  (who  was  completing  a  PhD  program)  gave  an 
introductory  presentation  explaining  the  background  for  and  scope  of  the  project,  and  was 
available  on  call  to  answer  questions  for  the  six  weeks  of  the  project.  The  graduate  student,  who 
was  studying  energy  yield  from  sweet  potatoes  as  an  MS  project  at  an  HBCU,  was  recommended 
by  his  preceptor,  who  had  earlier  worked  in  AFRL’s  labs.  The  teacher  and  high  school  student 
were  recommended  by  FSU-PC’s  PI,  who  is  also  a  long-time  member  of  the  board  of  the  Bay 
County  public  school  system.  An  Educational  Partnership  Agreement  was  in  place  between  the 
school  system  and  RXQ. 
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C.4.  The  Project 


The  team  was  given  six  weeks  to  complete  the  problem  posed;  all  necessary  tools,  equipment 
and  materials  were  available  on  site.  Once  assigned  their  task,  the  team  was  asked  to  prepare  by 
performing  literature  searches  to  determine  best  practices  to  collect  useful  data  and  to  familiarize 
themselves  with  the  test  protocols.  This  orientation  process  took  much  longer  than  expected  and 
put  the  group  behind  schedule  to  complete  their  experiments.  However,  once  the  experiments 
began,  they  were  able  to  collect  useful  data,  learn  from  their  experiments  and  mistakes  along  the 
way,  and  make  more  accurate  measurements  and  draw  conclusions  as  they  went  along. 

As  stated  above,  the  team’s  task  was  to  determine  the  best  available  concentration  and  size  of 
ceria  nanoparticles  (nanoceria)  to  add  to  an  algal  suspension  to  promote  growth.  The  team  ran 
experiments  to  determine  the  concentration  of  algae  by  measuring  algal  growth  with  time  in 
cultures  in  a  common  medium  containing  various  concentrations  of  nanoceria.  This  entailed 
developing  calibration  curves  that  would  1)  quantify  the  change  in  concentration  of  algae  and, 
separately,  the  concentration  of  nanoceria.  Both  were  measured  using  a  UV-visible 
spectrophotometer.  Figure  C-l  shows  algal  concentration  data  generated  by  the  team. 
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Figure  C-l.  Algal  Growth  after  Addition  of  Varying  Amounts  of  Ceria  Nanoparticles 
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The  team  also  used  light  and  electron  microscopy  to  visualize  effects  wrought  on  the  algal  cells 
by  differing  sizes  and  amounts  of  nanoceria.  Figure  C-2  and  Figure  C-3  are  light  and  scanning 
electron  micrographs  taken  by  the  team  to  illustrate  cellular  activity  and  cellular  interaction  with 
nanoceria. 

At  the  end  of  the  6  weeks  the  growth  experiments  were  incomplete  and  no  combustion  analyses 
had  been  started.  A  project  review  was  conducted  at  the  end  of  the  period  in  which  the  three 
participants  made  a  tag-team  presentation  of  the  project  and  attempted  to  answer  questions  from 
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a  small  panel  of  BCSS  and  FSU-PC  staff  and  AFRL  personnel.  No  written  materials  were 
submitted. 


Figure  C-2.  Light  Micrograph  of  Algae 


Figure  C-3.  Scanning  Electron  Micrograph 
of  Algae  and  Nanoceria 


C.5.  Results 

A  postmortem  interview  with  the  teacher  revealed  that  in  the  absence  of  effective  guidance  she 
had  assumed  control  of  the  lab  work  and  performed  most  of  it  herself  and  that  the  high-schooler 
had  not  engaged  as  deeply  as  had  been  hoped  for. 

A  postmortem  summary  was  also  provided  by  the  host  engineer,  who  devised  the  lab  task: 

The  team  consisted  of  three  diverse  individuals  with  vast  differences  in  scientific  background 
and  understanding,  which  posed  some  unique  challenges.  The  concept  for  the  structure  was  that 
the  graduate  student,  who  should  have  the  greatest  depth  of  understanding  of  the  field,  would 
lead  the  group,  develop  the  problem,  design  the  experiments,  and  guide  execution  of  the  plan  by 
giving  direction  to  the  other  team  members  and  integrating  results.  The  teacher,  bringing  a  basic 
understanding  of  science  and  learning  techniques,  was  to  help  execute  the  plan  and  work  with  the 
student  to  provide  a  mutual  learning  experience.  This  would  include  discovery  of  practical  uses 
of  science  and  engineering,  and  learning  new  concepts  and  the  importance  of  education  to  teach 
young  students.  Finally,  the  high  school  student  was  to  help  perfonn  basic  experimental 
procedures  and  preparatory  work  while  getting  first-hand  experience  in  a  laboratory  setting.  The 
small-group  environment  was  intended  to  draw  each  team  member  into  close  interaction  with  the 
other  to  promote  both  brainstorming  and  sharing  of  experiences  and  ideas. 

In  the  lab,  the  team  found  it  difficult  to  work  together  due  to  several  factors.  The  largest  problem 
facing  the  team — and  apparent  in  each  of  the  members — was  minimal  understanding  of  the 

15 

Approved  for  public  release;  distribution  unlimited. 


scientific  process.  This  created  a  problem  organizing  the  experiments,  executing  the  plan  and 
handling  data  to  interpret  results  and  design  next  experiments  to  conduct.  This  degraded 
motivation  to  keep  the  team  working  together  and  moving  forward  as  a  group,  and  led  to 
progressive  decreases  in  group  interaction  and  individuals  independently  pressing  forward, 
leaving  others  behind  “in  the  dust.” 

The  aim  of  this  program  was  to  allow  people  at  various  stages  of  development  with  various 
backgrounds  to  interact  to  achieve  a  common  objective.  In  this  case,  the  goal  was  only 
marginally  met.  A  preliminary  interview  with  the  participants  could  have  selected  a  better  team 
and  initial  selection  of  a  project  better  matched  to  the  capabilities  and  levels  of  understanding. 
This  could  also  be  enhanced  by  earlier  selection  of  the  team  members  and  more  preparation  time 
for  background  learning  and  interaction  with  each  other  and  project  leads. 

C.6.  Discussion 

The  three-person  team  was  allowed  to  work  at  the  Air  Force  Research  Lab,  Airbase 
Technologies  Division  (AFRL/RXQ)  in  the  Deployed  Energy  Research  facility  for  six  weeks, 
with  all  of  the  materials  and  equipment  that  were  needed  to  complete  their  designated  task.  The 
team  was  set  to  work  independently  establishing  their  own  structure  and  responsibilities  with 
minimal  assistance  from  on-site  staff.  Initial  guidance  was  provided  by  the  host  government 
engineer  on  the  desired  outcomes  of  the  task,  and  contracted  personnel  provided  minimal 
technical  assistance  and  support  of  materials. 

Through  some  combination  of  inadequate  training  and  shyness  the  graduate  student  provided  no 
leadership  to  the  team  (and,  during  interrogation  at  the  outbrief,  was  unable  to  describe  not  only 
the  project  but  also  the  technical  details  of  his  graduate  project  or  its  role  in  any  context).  In 
extreme  contrast,  the  teacher  dug  right  in  and  assumed  command  of  both  the  team  and  the 
project,  assigning  specific  tasks  to  the  other  two  and  providing  the  organization  that  drove  the 
effort.  The  high-schooler  engaged  only  part  time  in  the  project  and  was  distracted  by  late-night 
socializing  but,  by  the  outbrief  on  the  last  day  of  the  project,  had  acquired  enough  understanding 
and  experience  to  describe  accurately  those  parts  of  the  project  assigned  to  her  by  the  teacher. 

The  graduate  student  had  not.  Inefficiency  of  the  team  (which,  in  retrospect  and  as  it  evolved, 
should  have  been  addressed  by  active  intervention)  slowed  progress  to  an  extent  that  roughly  half 
the  tasks  of  the  project  were  accomplished.  The  dried  algal  masses  prepared  were  preserved  so 
combustion  analyses  could  be  performed  later  by  the  host  engineer’s  team — the  data  were  of  real 
value  to  his  project. 

C.7.  Conclusions 

The  design  of  this  exercise  is  sound  and  was  based  on  a  positive  history  of  summer  interns  at 
RXQ.  The  weakness  of  the  graduate  student  could  have  been  detected  with  a  preliminary  screen; 
if  an  opportunity  to  repeat  this  exercise  recurs  this  will  be  included  as  a  lesson  learned.  The 
scope  and  character  of  the  technical  project  were  appropriate  to  both  participant  skill  levels  and 
the  time  allotted.  The  principal  failings  were  in  the  execution  of  the  exercise,  and  responsibility 
lies  exclusively  at  the  top.  The  host  engineer  was  inexperienced  and  carried  the  idea  of 
independence  much  too  far — an  approach  of  guided  discovery  would  have  been  much  more 
effective.  A  preorientation  package  introducing  the  broad  area  of  the  topic  will  also  be  supplied 
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as  read-ahead  material  for  the  first  meeting.  The  PM  delegated  too  much  responsibility,  was 
physically  remote  (11  miles)  from  the  lab,  and  did  not  insist  on  regular  progress  reports.  The 
most  important  ingredient  missing  was  extensive  individual  interaction  with  technical 
professionals,  which  appears  at  all  grade  levels  to  be  the  single  most-effective  device  to  achieve 
effective  enrichment  at  any  level  and  in  any  context. 

An  opportunity  was  lost  to  raise  the  level  of  preparation  and  communication  skills  of  the 
graduate  student.  The  teacher  was  already  a  strong  performer  but  wavering  in  enthusiasm  in  the 
hostile  environment  of  public  education  in  Florida;  the  experience  of  the  summer  appears  both  to 
have  broadened  her  perspective  into  the  scientific  enterprise  and  bolstered  her  resolve  to  stay  on 
the  job — an  unintended  benefit  to  STEM  education  where  it  needs  help  the  most.  That  the  senior 
later  changed  her  announced  major  from  fashion  design  to  coastal  biology  may  be  the  single 
tangible  product  of  the  exercise,  as  the  change  may  be  a  first  step  toward  an  eventual  career  in  a 
harder  science  or  medicine.  It  also  suggests  that  similar  exposure  of  students  selected  by  science 
teachers  to  an  effectively  supervised,  hands-on  lab  project  might  be  an  effective  device  to  steer 
talent  into  STEM  training. 

As  in  the  project  in  the  Title  1  classrooms,  it  appears  that  most  if  not  all  of  the  operational 
missteps  available  and  not  related  to  safety  were  made  in  the  course  of  this  exercise.  Assuming 
that  the  lessons  learned  are  assimilated  and  addressed  during  the  next  iteration,  there  is  every 
reason  to  expect  that  the  concept  will  be  applied  with  active  but  minimally  intrusive  supervision 
by  experienced  personnel  with  appropriate  technical  qualifications,  and  that  the  enrichments 
intended  will  be  achieved. 
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LIST  OF  SYMBOLS,  ABBREVIATIONS  AND  ACRONYMS 


AFB 

Air  Force  Base 

AFRL 

Air  Force  Research  Laboratory 

BCSS 

Bay  County  (Florida)  public  school  system 

EPA 

Educational  Partnership  Agreement 

FSU-PC 

Florida  State  University,  Panama  City  campus 

PLTW 

Project  Lead  the  Way 

PM 

program  manager 

RXQ 

Airbase  Technologies  Division 

SME 

subject  matter  expert 

STEM 

Science,  Engineering,  Technology  and  Mathematics 

SY 

school  year 
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